The marginalization paradox and the formal Bayes' law

It has recently been shown that the marginalization paradox (MP) can be resolved by interpreting improper inferences as probability limits. The key to the resolution is that probability limits need not satisfy the formal Bayes' law, which is used in the MP to deduce an inconsistency. In this paper, I explore the differences between probability limits and the more familiar pointwise limits, which do imply the formal Bayes' law, and show how these differences underlie some key differences in the interpretation of the MP.Comment: Presented at Maxent 2007, Saratoga Springs, NY, July 200

Information and Entropy

What is information? Is it physical? We argue that in a Bayesian theory the notion of information must be defined in terms of its effects on the beliefs of rational agents. Information is whatever constrains rational beliefs and therefore it is the force that induces us to change our minds. This problem of updating from a prior to a posterior probability distribution is tackled through an eliminative induction process that singles out the logarithmic relative entropy as the unique tool for inference. The resulting method of Maximum relative Entropy (ME), which is designed for updating from arbitrary priors given information in the form of arbitrary constraints, includes as special cases both MaxEnt (which allows arbitrary constraints) and Bayes' rule (which allows arbitrary priors). Thus, ME unifies the two themes of these workshops -- the Maximum Entropy and the Bayesian methods -- into a single general inference scheme that allows us to handle problems that lie beyond the reach of either of the two methods separately. I conclude with a couple of simple illustrative examples.Comment: Presented at MaxEnt 2007, the 27th International Workshop on Bayesian Inference and Maximum Entropy Methods (July 8-13, 2007, Saratoga Springs, New York, USA

Origins of the Combinatorial Basis of Entropy

The combinatorial basis of entropy, given by Boltzmann, can be written $H = N^{-1} \ln \mathbb{W}$, where $H$ is the dimensionless entropy, $N$ is the number of entities and $\mathbb{W}$ is number of ways in which a given realization of a system can occur (its statistical weight). This can be broadened to give generalized combinatorial (or probabilistic) definitions of entropy and cross-entropy: $H=\kappa (\phi(\mathbb{W}) +C)$ and $D=-\kappa (\phi(\mathbb{P}) +C)$, where $\mathbb{P}$ is the probability of a given realization, $\phi$ is a convenient transformation function, $\kappa$ is a scaling parameter and $C$ an arbitrary constant. If $\mathbb{W}$ or $\mathbb{P}$ satisfy the multinomial weight or distribution, then using $\phi(\cdot)=\ln(\cdot)$ and $\kappa=N^{-1}$, $H$ and $D$ asymptotically converge to the Shannon and Kullback-Leibler functions. In general, however, $\mathbb{W}$ or $\mathbb{P}$ need not be multinomial, nor may they approach an asymptotic limit. In such cases, the entropy or cross-entropy function can be {\it defined} so that its extremization ("MaxEnt'' or "MinXEnt"), subject to the constraints, gives the ``most probable'' (``MaxProb'') realization of the system. This gives a probabilistic basis for MaxEnt and MinXEnt, independent of any information-theoretic justification. This work examines the origins of the governing distribution $\mathbb{P}$.... (truncated)Comment: MaxEnt07 manuscript, version 4 revise

From Information Geometry to Newtonian Dynamics

Newtonian dynamics is derived from prior information codified into an appropriate statistical model. The basic assumption is that there is an irreducible uncertainty in the location of particles so that the state of a particle is defined by a probability distribution. The corresponding configuration space is a statistical manifold the geometry of which is defined by the information metric. The trajectory follows from a principle of inference, the method of Maximum Entropy. No additional "physical" postulates such as an equation of motion, or an action principle, nor the concepts of momentum and of phase space, not even the notion of time, need to be postulated. The resulting entropic dynamics reproduces the Newtonian dynamics of any number of particles interacting among themselves and with external fields. Both the mass of the particles and their interactions are explained as a consequence of the underlying statistical manifold.Comment: Presented at MaxEnt 2007, the 27th International Workshop on Bayesian Inference and Maximum Entropy Methods (July 8-13, 2007, Saratoga Springs, New York, USA

Updating Probabilities with Data and Moments

We use the method of Maximum (relative) Entropy to process information in the form of observed data and moment constraints. The generic "canonical" form of the posterior distribution for the problem of simultaneous updating with data and moments is obtained. We discuss the general problem of non-commuting constraints, when they should be processed sequentially and when simultaneously. As an illustration, the multinomial example of die tosses is solved in detail for two superficially similar but actually very different problems.Comment: Presented at the 27th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering, Saratoga Springs, NY, July 8-13, 2007. 10 pages, 1 figure V2 has a small typo in the end of the appendix that was fixed. aj=mj+1 is now aj=m(k-j)+

Information Geometry and Chaos on Negatively Curved Statistical Manifolds

A novel information-geometric approach to chaotic dynamics on curved statistical manifolds based on Entropic Dynamics (ED) is suggested. Furthermore, an information-geometric analogue of the Zurek-Paz quantum chaos criterion is proposed. It is shown that the hyperbolicity of a non-maximally symmetric 6N-dimensional statistical manifold M_{s} underlying an ED Gaussian model describing an arbitrary system of 3N non-interacting degrees of freedom leads to linear information-geometric entropy growth and to exponential divergence of the Jacobi vector field intensity, quantum and classical features of chaos respectively.Comment: 7 pages, presented at MaxEnt2007, the 27th International Workshop on Bayesian Inference and Maximum Entropy Methods, Saratoga (NY, USA) (July-2007

Lessons about likelihood functions from nuclear physics

Least-squares data analysis is based on the assumption that the normal (Gaussian) distribution appropriately characterizes the likelihood, that is, the conditional probability of each measurement d, given a measured quantity y, p(d | y). On the other hand, there is ample evidence in nuclear physics of significant disagreements among measurements, which are inconsistent with the normal distribution, given their stated uncertainties. In this study the histories of 99 measurements of the lifetimes of five elementary particles are examined to determine what can be inferred about the distribution of their values relative to their stated uncertainties. Taken as a whole, the variations in the data are somewhat larger than their quoted uncertainties would indicate. These data strongly support using a Student t distribution for the likelihood function instead of a normal. The most probable value for the order of the t distribution is 2.6 +/- 0.9. It is shown that analyses based on long-tailed t-distribution likelihoods gracefully cope with outlying data.Comment: presented at 27th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering (Maxent 2007), 10 pages, 12 figure

On Shannon-Jaynes Entropy and Fisher Information

The fundamentals of the Maximum Entropy principle as a rule for assigning and updating probabilities are revisited. The Shannon-Jaynes relative entropy is vindicated as the optimal criterion for use with an updating rule. A constructive rule is justified which assigns the probabilities least sensitive to coarse-graining. The implications of these developments for interpreting physics laws as rules of inference upon incomplete information are briefly discussed.Comment: Presented at the 27th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering, Saratoga Springs, NY, July 8-13, 200